System and method for predictive fusion

ABSTRACT

An image fusion system provides a predicted alignment between images of different modalities and synchronization of the alignment, once acquired. A spatial tracker detects and tracks a position and orientation of an imaging device within an environment. A predicted pose of an anatomical feature can be determined, based on previously acquired image data, with respect to a desired position and orientation of the imaging device. When the imaging device is moved into the desired position and orientation, a relationship is established between the pose of the anatomical feature in the image data and the pose of the anatomical feature imaged by the imaging device. Based on tracking information provided by the spatial tracker, the relationship is maintained even when the imaging device moves to various positions during a procedure

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/970,973, filed on May 4, 2018. Application Ser. No. 15/970,973 claimspriority to U.S. Provisional Application Ser. No. 62/501,329, filed onMay 4, 2017. The entireties of the aforementioned applications areherein incorporated by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This application relates generally to image registration and fusion and,more particularly, to systems and methods for facilitating imageregistration and fusion via spatial tracking of an imaging device.

2. Description of Related Art

Image fusion generally relates to combining information from differentimages into a single, composite image. In medical imaging, for instance,fusion can involve registering and combining different images, in somemanner, to generate a composite image. The composite image can provideimproved image quality or enhance usability of the images for diagnosis,treatment planning and assessment, tracking disease progression, etc. Inmedical imaging, the two or more images fused can be of the same imagingmodality or different imaging modalities. Multiple images of the samemodality may be fused to ascertain disease progression or treatmentefficacy. Images of different modalities can be combined to leveragebenefits of the differing modalities or for convenience.

For instance, magnetic resonance imaging (MRI) provides good soft tissuecontrast. Thus, MRI provides relatively easy differentiation of lesionsor other abnormalities from healthy tissue. Accordingly, MRI performswell for detection and planning. With image-guided procedures, MRI canbe inconvenient due to cost and non-portability of the imaging machine.For example, taking a biopsy of a prostate is often guided byultrasound, which is portable and provides high spatial resolution.Compared to MRI, however, ultrasound provides less tissuediscrimination. Thus, an MRI-ultrasound fusion can combine informationfrom the respective modalities to improve execution of the image-guidedprocedure.

BRIEF SUMMARY OF THE INVENTION

A simplified summary is provided herein to help enable a basic orgeneral understanding of various aspects of exemplary, non-limitingembodiments that follow in the more detailed description and theaccompanying drawings. This summary is not intended, however, as anextensive or exhaustive overview. Instead, the sole purpose of thesummary is to present some concepts related to some exemplarynon-limiting embodiments in a simplified form as a prelude to the moredetailed description of the various embodiments that follow.

In various, non-limiting embodiments, an image fusion system provides apredicted alignment between images of different modalities andsynchronization of the alignment, once acquired. A spatial trackerdetects and tracks a position and orientation of an imaging devicewithin an environment. The imaging device, for instance, is a suitabledevice for intra-procedural imaging. Based on image data of a differentmodality to the imaging device, a predicted pose of an anatomicalfeature can be determined with respect to a desired position andorientation of the imaging device. When the imaging device is moved intothe desired position and orientation, a relationship is establishedbetween the pose of the anatomical feature in the image data and thepose of the anatomical feature imaged by the imaging device. Based ontracking information provided by the spatial tracker, the relationshipis maintained even when the imaging device moves to various positionsduring a procedure

These and other embodiments are described in more detail below.

BRIEF DESCRIPTION OF THE DRAWING

Various non-limiting embodiments are further described with referencethe accompanying drawings in which:

FIG. 1 is a block diagram of an exemplary, non-limiting embodiment foran image fusion system according to one or more aspects;

FIG. 2 is a schematic block diagram of an exemplary, non-limitingembodiment for a computing device associated with the image fusionsystem of FIG. 1;

FIG. 3 is a schematic block diagram of an exemplary, non-limitingembodiment for medical image fusion according to one or more aspects;

FIG. 4 is a flow diagram of an exemplary, non-limiting method forpredicting a fusion between at least two image modalities;

FIG. 5 is a flow diagram of an exemplary, non-limiting method forgenerating a fusion between a live image and a previously acquiredimage;

FIG. 6 is an exemplary image having an contour overlaid to indicate apredicted pose of an anatomical feature with respect to a desiredposition and orientation of an imaging device;

FIG. 7 is an exemplary image captured by the image device subsequent toan operator positioning the imaging device to align the anatomicalfeature as imaged with the overlaid contour;

FIG. 8 is an exemplary image captured following movement of the imagingdevice and after the overlaid contoured is updated responsive to themovement of the image device; and

FIG. 9 is a block diagram representing an exemplary, non-limitingcomputing system or operating environment in which one or more aspectsof various embodiments described herein can be implemented.

DETAILED DESCRIPTION OF THE INVENTION General Overview

As discussed in the background, medical image fusion can leveragestrengths of different imaging modalities and generate combinedinformation having a wide array of applications. For example, a fusionof MRI and ultrasound images of a prostrate can provide effectiveintra-procedural images with accurate identification of anatomicalfeatures. Due to many factors, however, the respective orientations ofthe prostate in MRI images and intra-procedural ultrasound images canvary significantly. These significant differences result incomputationally difficult and imperfect registrations between the twomodalities.

In various, non-limiting embodiments, a system and associated methodsare provided for image fusion. Based on a live image provided by aspatially tracked imaging device, it can be determined whether theimaging device aligns to a desired position and orientation with respectto an anatomical feature. In particular, the desired position andorientation can be designated based on feature information associatedwith previously acquired image data, which may be a different modalitythan the live image. The feature information specifies a position andorientation of the anatomical feature in the previously acquired imagedata. This image data and feature information can be transformed so asto indicate a pose of the feature as viewed from the perspective of theimaging device in the desired position and orientation.

Once aligned to the desired position and orientation, the relationshipbetween the live image and the previously acquired image data can beestablished. When locked, tracking information from a spatial trackercan inform processing of the previously acquired image data.Specifically, the image data and/or feature information is transformedin response to movement of the imaging device to maintain alignment. Forinstance, a spatial transformation between a current spatial positionand a reference spatial position (i.e., the desired position andorientation) can be determined based, at least in part, on the trackinginformation. A corresponding transformation, based on the spatialtransformation, can subsequently be applied to the previously acquiredimage and/or feature information.

In one embodiment, a system is provided that includes a processorcoupled to memory storing computer-executable instructions. Whenexecuted by the processor, the instructions configure the processor to:obtain feature information indicative of a pose of a feature in a firstimage of a first modality; obtain tracking information, with at leastthree degrees of freedom, from a spatial tracker configured to identifya position and orientation of a handheld imaging device producing imagesof a second modality; and determine when a pose of the imaging devicematches a predetermined pose suitable to produce a second image of thefeature having a pose that matches a reference pose of the feature basedon at least one of the feature information or the tracking information.According to one example, the feature information specifies anorientation of the feature in the first image relative to a givenorientation of the imaging device. For instance, the feature informationcan include a contour of the feature in the first image.

The processor can be further configured to update responsive to movementof the imaging device, an indication of the pose of the feature in thesecond image produced by the imaging device based on at least one of thetracking information or the feature information. The processor can befurther configured to obtain a relationship between the pose of theimaging device in space and a pose of an imaging plane of the imagingdevice and update the indication of the pose of the feature in thesecond image based on the relationship. In addition, a spatialrelationship between a current pose of the imaging device and areference pose of the imaging device is also determined, for example,based on the tracking information. The reference pose of the imagingdevice can correspond to the predetermined pose mentioned above, forinstance. To update the indication of the pose of the feature, theprocessor can be configured to reslice the first image along a virtualimaging plane. The processor can be configured to determine the virtualimaging plane based on the tracking information. In addition, theprocessor can be configured to interpolate image data of the first imagecorresponding to the virtual imaging plane. In yet another example, theprocessor can be configured to identify a transform between the firstimage and the second image and to apply the transform to the featureinformation.

Still further, the processor can be configured to display a live imageacquired from the imaging device. The processor can also be configuredto display an overlay on the live image based on the featureinformation. In addition, the processor can be configured to update theoverlay based on the indication of the pose of the feature in the secondimage as acquired by the imaging device.

According to another aspect, a method is described. The method includesacquiring a live image of a first modality from an imaging device. Inaddition, the method can include acquiring tracking information from aspatial tracker that indicates a position and orientation of the imagingdevice. Further, the method can also include determining when theimaging device is in a desired pose relative to a feature in the liveimage thereby establishing a relationship between a pose of the imagingdevice to a pose of the feature in a second image of a second modality,wherein determining the imaging device is in the desired pose is basedon at least one the tracking information or feature informationindicative of the pose of the feature in the second image.

According to an example, the method can include updating an indicationof the feature in the live image, responsive to movement of the imagingdevice, based on at least one of the tracking information or theestablished relationship. The indication of the feature can be initiallygenerated based on feature information indicating the pose of thefeature in the second image with respect to the desired pose of theimaging device relative to the feature. The feature information can be acontour, for instance. Updating the indication of the feature caninclude reslicing the second image along a virtual imaging plane,wherein the virtual imaging plane is determined based on the trackinginformation. Reslicing, in turn, can include interpolating image data ofthe second image corresponding to the virtual imaging plane. In anotherexample, updating the indication of the feature can include determininga transform between the desired pose of the imaging device and theposition and orientation of the imaging device provided by the trackinginformation, and applying the transform to the indication of thefeature.

In yet another embodiment, a computer-readable storage medium isdescribed. The computer-readable storage medium storescomputer-executable instructions that, when executed by a processor,configure the processor to: obtain feature information indicative of afeature in a first image of a first modality and a pose of the featurein the first image with respect to a desired pose of an imaging deviceconfigured to produce images of a second modality; obtain trackinginformation from a spatial tracker configured to track a position andorientation of the imaging device in space with at least three degreesof freedom; and determine when the imaging device is aligned with thedesired pose based on at least one of a live image produced by theimaging device, the feature information, or the tracking information.

In an example, the computer-executable instructions stored on thecomputer-readable storage medium further configure the processor toupdate an indication of the feature in the live image based on at leastone of the tracking information or the feature information. Forinstance, the medium can include instructions that configure theprocessor to reslice the first image along a virtual imaging planedetermined based on the tracking information and interpolate the featureinformation along the virtual imaging plane to generate an updatedindication of the pose of the feature in the live image. In anotherexample, the computer-executable instructions stored on thecomputer-readable storage medium further configure the processor todetermine a transform between the desired pose and the position andorientation of the imaging device provided by the tracking information;and to apply the transform to the feature information to update theindication of the pose of the feature in the live image. In addition,the medium further stores instructions that configure the processor todisplay an initial indication of the pose of the feature in the liveimage, wherein the initial indication is generated based on the featureinformation and provides a guide to align the imaging device with thedesired pose.

An overview of some embodiments of predictive fusion has been presentedabove. As a roadmap for what follows next, predictive image fusion isdescribed in more detail. Afterwards, an exemplary computing device andcomputing environment, in which such embodiments and/or featuredescribed above can be implemented, are described. The above notedfeatures and embodiments will be described with reference to thedrawings, wherein like reference numerals are used to refer to likeelements throughout.

Predictive Image Registration and Fusion

As mentioned above, in various embodiments, a fusion of images ofdifferent modalities can be determined and alignment between the imagescan be locked even with an imaging device providing live images andcapable of freeform movement in space.

FIG. 1 shows a block diagram illustrating an exemplary, non-limitingembodiment for an image fusion system 100. As shown, image fusion system100 can include a computing device 110, an imaging device 120, and aspatial tracker 130. The computing device can include a processor andvarious computer-readable storage media (e.g., volatile andnon-volatile). The computer-readable storage media can storecomputer-executable instructions implementing at least a portion offunctional modules, such as fusion engine 112, described herein. Whenthe computer-executable instructions are executed by the processor, theimage fusion system 100 is thus configured to perform the operationsdescribed herein.

Computing device 110 can further include various hardware devices (notshown) to implement portions of fusion engine 112. For instance,computing device 110 can include a graphics device having a graphicsprocessing unit (GPU), dedicated memory, and/or hardware interfaces tocouple the graphics device to a display. Moreover, computing device 110can include physical hardware ports and/or wireless interfaces (e.g.,Bluetooth, wireless USB, etc.) to couple computing device 110 to variousdevices of image fusion system 100, such as, but not limited to, imagingdevice 120 and spatial tracker 130.

Imaging device 120, as shown, can include an imaging probe 122 and animage processor 124. In an aspect, imaging device 120 can be arelatively inexpensive and portable device suitable for intra-proceduralimaging, such as an ultrasound imaging device. Nonetheless, it is to beappreciated that features and aspects described and claimed herein arenot limited to ultrasound applications and can be readily adapted foruse with other imaging modalities.

In the ultrasound example, imaging probe 122 can include one or moretransducer arrays configures to emit ultrasonic pulses and receiveechoes. The echoes can be converted to electrical signals and providedto image processor 124 to generate an ultrasound image. While the imageprocessor 124 is shown separate from computing device 110, it is to beappreciated that processing of echo signals can be performed by computerdevice 110. For example, a separate software application or a module offusion engine 112 can be configured to process signals from imagingprobe 122. Moreover, while imaging device 120 is shown as including boththe imaging probe 122 and the image processor 124, the term “imagingdevice” as utilized herein can refer to all components that collectivelyinteroperate to generate an image or, depending on context, can refer tothe portion housing the transducer arrays (i.e. the probe). Forinstance, it is to be appreciated that, when described in connectionwith spatial tracking, the term “imaging device” means the portion of anoverall imaging apparatus that is capable of manipulation in order todirect or target what is ultimately imaged.

In an aspect, a sensor 132 can be coupled to or integrated with imagingprobe 122. Sensor 132 cooperates with spatial tracker 130 to generatetracking information indicative of a position and orientation of theimaging probe 122 in space. According to an example, spatial tracker 130can be an electromagnetic (EM) tracking system comprising an EM sourcethat generates a EM field, which establishes a three-dimensional frameof reference. Pursuant to this example, sensor 132 can include inductioncoils or other devices, which are orthogonally aligned, and generatesignals indicative of strength of the received EM field. The signalsenable determination of a position and orientation of the sensor 132 inthe three-dimensional frame of reference established by the EM source.The signals can be transmitted to the computing device 110, via a wiredor wireless connection, for processing and position determination. Inthe EM tracking system, for example, the signals are typicallycommunicated by sensor 132. Alternatively, however, the signals can bereceived by spatial tracker 130 and, subsequently, forwarded tocomputing device 110, with pre-processing or in a raw format. Further,while the above examples contemplate an EM-based positioning system, itis to be appreciated that other trackers can be utilized. For example,spatial tracker 130 can be an accelerometer/gyroscopic-based tracker, aninfrared tracker, an optical tracker, an acoustic tracker, a lasertracker, an RF-based tracker, or substantially any other type of spatialtracking and positioning system capable of identifying a position andorientation of a sensor within a frame of reference, such as but notlimited to, mechanical spatial tracking (e.g. a probe stepper organtry).

In a further aspect, sensor 132 is coupled to or integrated with imagingprobe 122 so as to establish a known relationship between the locationof sensor 132 relative to imaging planes of probe 122. Accordingly,based on this relationship, the position and orientation of the imagingplanes (and generated images) within the spatial frame of referenceprovided by spatial tracker 130 are also known. The relationship betweenthe location of sensor 132 and the imaging planes can be determined bymeasuring the relationship (i.e., explicitly designed by a probemanufacturer) or by calibration. By way of illustration, one techniqueto calibrate or establish the known relationship between the location ofsensor 132 in the spatial frame of reference and the imaging planesinvolves imaging a control object. For example, an object (e.g., anintersection point of several threads) can be imaged in a bucket ofwater from a variety of positions and/or orientations. A spatialtransform is then determined, which solves the relationship betweenpixels in the images identified as the object and the sensor's positionin the spatial frame of reference. For instance, the spatial transformcan be the solution to a system of linear equations including severalunknown variables (translations and rotations in space) and the multipleimages acquired of the object. The system of linear equations can beoverdetermined such that the number of images acquired is greater thanthe number of unknowns. It is to be appreciated that the above techniqueis merely illustrative and other calibration techniques can be utilized.

Fusion engine 112 utilizes this relationship to facilitate fusionbetween images produced by imaging device 120 and image data 140, whichcan include previously acquired image data of a different modality, forexample. To illustrate, consider an image-guided biopsy of the prostate.Image data 140 can include imaging of the prostate in a differentmodality from that produced by the imaging device 120. The modality ofimage data 140 may provide better tissue discrimination capabilities sothat the prostate can be readily identified and healthy tissue of theprostate can be differentiated from abnormal tissue.

The location or pose of the prostate in the previously acquired image aswell as the location or pose of abnormal tissue can be specified byfeature information. In an example, feature information can include acontour of the prostate and a separate contour for the abnormal tissue.In general, however, the term “feature information” relates to imagingor other data that specifies a pose of a feature or object in medicalimages. As utilized herein, the term “pose” refers to a position andorientation of an object in a given frame of reference, which can bedefined relative to another object. By way of example, the pose of afeature in an image relates to the position and orientation of thefeature as shown in the image or within the imaging space or volume. Thepose of an imaging device relates to the position and orientation of theimaging device relative to an identified reference (e.g., a feature, atracking volume, etc.) and can also refer to an orientation of animaging plane of the imaging device relative to the same identifiedreference.

With the pose of the feature (e.g., prostate) determined in image data140, fusion engine 112 can determine a reference position andorientation (or pose) for the imaging device 120 relative to thefeature. The reference position can be indicated on a display as, forexample, an outline of the feature from a viewpoint of an imaging devicein the reference pose. More particularly, an imaging device in thereference pose defines an imaging plane. Image data 140 and/or featureinformation is resampled along the imaging plane to derive the outline.That is, voxel data or contour data intersecting the imaging plane isutilized to generate the outline, or other indication, of the feature.

Imaging device 120 can generate a live image, which can be displayed bycomputing device 110. Fusion engine 112 is configured to overlay theoutline of the feature on the live image to facilitate alignment. Forexample, FIG. 6 depicts an exemplary initial image captured by theimaging device 120 and displayed with the outline generated by thefusion engine 112 as described above. An operator manipulates theimaging device 120 until the feature, as shown in the live image,corresponds to the outline. For example, FIG. 7 depicts an exemplaryimage after the operator manipulates the image device 120 to align theimaged feature to the outline.

Once a correspondence is achieved, fusion engine 112 can establishes arelationship between the pose of the imaging device 120 in space (i.e.the frame of reference provided by spatial tracker 130) and the pose ofthe feature in image data 140. Fusion engine 112 employs thisrelationship to lock in a fusion between the live image and the imagedata 140. Particularly, the fusion engine 112 maintains alignmentbetween the displayed feature information (e.g., outline) and the poseof the feature in the live image. For example, when the operator movesthe imaging device 120, image data 140 is processed and transformed toprovide a correspondingly updated outline of the feature displayed overthe changing live image (i.e. updates the fusion). For instance, FIG. 8depicts an exemplary image captured by the imaging device 120 after theoperator moves (e.g. tilts) the imaging device 120. In FIG. 8, theoutline of the feature is updated in response to the movement of theimaging device to maintain correspondence.

Spatial tracker 130 generates tracking information indicating the poseof the imaging device 120 in space. Fusion engine 112, based on thetracking information, determines a position and orientation of acorresponding imaging plane and reslices the image data 140 along thisimaging plane. Feature information (e.g., contours) can also be reslicedalong the imaging plane to generate the updated outline. Thus, as theoperator moves the imaging probe 122 and the live image changes, fusionengine 112 responsively updates feature information derived from theimage data 140 to provide a composite image based on the live image andimage data 140.

If a patient shifts position, with or without a corresponding movementof the imaging device 120, the relationship between the imaging planeand the feature may change. Accordingly, the relationship (i.e., aninitial alignment) may need updated. In another aspect, the computingdevice 110 may enable the operator to manually adjust the outline orcontour to correct the relationship. Alternatively, the fusion engine112 can relock the relationship. For example, the fusion engine 112 candetermine a new reference pose and identify when the features as imagedby the imaging device 120 corresponds to the new reference pose.

FIG. 2 illustrates a schematic block diagram of an exemplary,non-limiting embodiment for a computing device 110 associated with imagefusion system 100 of FIG. 1. As shown in FIG. 2, computing device 110includes one or more processor(s) 202 configured to executedcomputer-executable instructions such as instructions composing fusionengine 112. Such computer-executable instructions can be stored on oneor more computer-readable media including non-transitory,computer-readable storage media such as storage 208. For instance,storage 208 can include non-volatile storage to persistently storefusion engine 112 and/or data 212 (e.g., image data, featureinformation, tracking information, captured image data, configurationinformation, working data, etc.). Storage 208 can also include volatilestorage that stores fusion engine 112 and other data 212 (or portionsthereof) during execution by processor 202.

Computing device 110 includes a communication interface 206 to couplecomputing device 110 to various remote systems (e.g. an image datastore, an imaging apparatus, etc.). Communication interface 206 can be awired or wireless interface including, but not limited, a WiFiinterface, an Ethernet interface, a fiber optic interface, a cellularradio interface, a satellite interface, etc. An I/O interface 210 isalso provided to couple computing device 210 to various input and outputdevices such as displays, touch screens, keyboards, mice, touchpads,etc. By way of example, I/O interface 210 can include wired or wirelessinterfaces such as, but not limited to, a USB interface, a serialinterface, a WiFi interface, a short-range RF interface (Bluetooth), aninfrared interface, a near-field communication (NFC) interface, etc.Also shown in a peripheral interface 204 to couple computing device 110,wired or wirelessly, to various peripherals utilized by fusion engine112. For example, peripheral interface 204 can couple computing device110 to sensor 132 to receive signals, imaging device 120 to receive liveimages, imaging probe 122 to receive raw signals for processing, and/orspatial tracker 130 to receive tracking information.

Turning now to FIG. 3, a block diagram of an exemplary, non-limitingfusion engine 112 is depicted. As shown in FIG. 3, fusion engine 112 caninclude various functional modules implemented by computer-executableinstructions. The modules can include an initialization module 302, async module 304, a fusion module 306, and a display module 308.

Initialization module 302 receives user input 314 to configure fusionengine 112 to perform a medical image fusion as disclosed herein. Forexample, initialization module 302 can load image data 316 from storage,local or remote, based on a selection included in user input 314.Initialization module 302 can also establish a reference pose for animaging device, which can be a default pose or a pose provided in userinput 314. As mentioned above, the reference pose can be a desired posethat can be an initial target pose for the imaging device in order toachieve image fusion in accordance with one or more aspects herein.Based on the reference pose, initialization module 302 can process imagedata 316 and feature information 318 to generate initial fusioninformation (e.g., display data or the like) that is combined with liveimage data 322 by display module 308 to generate composite image 324displayable on a display.

Sync module 304 determines when the imaging device aligns with thereference pose and locks in a relationship between a pose of imagingdevice in space and a position and orientation of the feature in image316 as provided by feature information 318. Sync module 304 candetermine alignment based on user input 314. For example, when initialfusion information composited with live image data 322 visually showsalignment between the feature in the live image and the feature fromimage data 316, an operator can provide input. Alternatively, aregistration engine or other image processing modules can evaluatecomposite image 324, image data 316, feature information 318, and/orlive image data 322 to computationally determine alignment.

Once the relationship is established, fusion module 306 maintains thealignment. For instance, fusion module 306 includes a pose determinationmodule 310 that determines a pose of the image device based on trackinginformation 320 from a spatial tracker. Transform module 312 processesimage data 316 and/or feature information 318 to update the fusioninformation that is combined with live image data 322 to generatecomposite image 324. As described above, transform module 312 canreslice image data 316 and/or feature information 318 according to avirtual image plane corresponding to an imaging plane of the imagingdevice in the pose specified by tracking information 320. In anotherexample, a transform can be determined based on tracking information 320and the determined transform can be directly applied to the fusioninformation composited with live image data 322.

FIG. 4 illustrates a flow diagram of an exemplary, non-limiting methodfor predicting a fusion between at least two image modalities. Themethod of FIG. 4 can be performed, for example, by image fusion system100 and/or fusion engine 112 executed on computing device 110 asdescribed previously. At 400, a position and orientation of a feature ina first image of a first modality is identified. For example, a contourcan be defined that indicates the feature in the first image. At 402,tracking information specifying a position and orientation of an imagingdevice of a second modality is obtained. The tracking information can beacquired by a spatial tracking or other positioning system. At 404, adetermination is made when the imaging device is placed in apredetermined pose relative to the feature. The determination can bebased on the tracking information, feature information (e.g., contours),image data of the first image, or user input. At 406, first image dataand/or associated contours are transformed responsive to the trackinginformation to maintain alignment between the first image and imagesproduced by the imaging device.

Referring now to FIG. 5, a flow diagram of an exemplary, non-limitingmethod for generating a fusion between a live image and a previouslyacquired image is illustrated. The method of FIG. 5 can be performed,for example, by image fusion system 100 and/or fusion engine 112executed on computing device 110 as described above. At 500, a liveimage of a first modality is acquired from an imaging device. At 502, anindication of a feature, from a second image of a different modality, isdisplayed over the live image. At 504, a relationship is establishedbetween the live image and the second image when it is determined thatthe imaging device is in a desired posed relative to the feature. At506, tracking information is acquired. The tracking informationspecifies a position and orientation of the imaging device in areference space. At 508, the second image is processed based on thetracking information to update the indication of the feature andpossibly other features, displayed over the live image.

The exemplary embodiments described above are presented in the contextof image fusion. It is to be appreciated that these concepts can beextended to other contexts, such as image correlation. For instance, asopposed to combining or compositing (i.e. overlaying) portions ofimages, the images, or portions thereof, can be displayed side-by-sideor in another arrangement for comparison.

Elemplary Computing Device

As mentioned, advantageously, the techniques described herein can beapplied to any device where it is desirable to provide predictive fusionand live tracking of images of different modalities. It can beunderstood, therefore, that handheld, portable and other computingdevices and computing objects of all kinds are contemplated for use inconnection with the various embodiments of a registration visualizationsystem. Accordingly, the below general purpose computer described belowin FIG. 9 is but one example of a computing device.

Embodiments can partly be implemented via an operating system, for useby a developer of services for a device or object, and/or includedwithin application software that operates to perform one or morefunctional aspects of the various embodiments described herein. Softwaremay be described in the general context of computer-executableinstructions, such as program modules, being executed by one or morecomputers, such as client workstations, servers or other devices. Thoseskilled in the art will appreciate that computer systems have a varietyof configurations and protocols that can be used to communicate data,and thus, no particular configuration or protocol is consideredlimiting.

FIG. 9 thus illustrates an example of a suitable computing systemenvironment 900 in which one or aspects of the embodiments describedherein can be implemented, although as made clear above, the computingsystem environment 900 is only one example of a suitable computingenvironment and is not intended to suggest any limitation as to scope ofuse or functionality. In addition, the computing system environment 900is not intended to be interpreted as having any dependency relating toany one or combination of components illustrated in the exemplarycomputing system environment 900.

With reference to FIG. 9, an exemplary device for implementing one ormore embodiments includes a general purpose computing device in the formof a computer 910. Components of computer 910 may include, but are notlimited to, a processing unit 920, a system memory 930, and a system bus922 that couples various system components including the system memoryto the processing unit 920.

Computer 910 typically includes a variety of computer readable media andcan be any available media that can be accessed by computer 910. Thesystem memory 930 may include computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) and/orrandom access memory (RAM). By way of example, and not limitation,system memory 930 may also include an operating system, applicationprograms, other program modules, and program data. According to afurther example, computer 910 can also include a variety of other media(not shown), which can include, without limitation, RAM, ROM, EEPROM,flash memory or other memory technology, compact disk (CD) ROM, digitalversatile disk (DVD) or other optical disk storage, or other tangibleand/or non-transitory media which can be used to store desiredinformation.

A user can enter commands and information into the computer 910 throughinput devices 940. A monitor or other type of display device is alsoconnected to the system bus 922 via an interface, such as outputinterface 950. In addition to a monitor, computers can also includeother peripheral output devices such as speakers and a printer, whichmay be connected through output interface 950.

The computer 910 may include a network interface 960 so as to operate ina networked or distributed environment using logical connections to oneor more other remote computers, such as remote computer 970. The remotecomputer 970 may be a personal computer, a server, a router, a networkPC, a peer device or other common network node, or any other remotemedia consumption or transmission device, and may include any or all ofthe elements described above relative to the computer 910. The logicalconnections depicted in FIG. 9 include a network 971, such local areanetwork (LAN) or a wide area network (WAN), but may also include othernetworks/buses. Such networking environments are commonplace in homes,offices, enterprise-wide computer networks, intranets and the Internet.

As mentioned above, while exemplary embodiments have been described inconnection with various computing devices and network architectures, theunderlying concepts may be applied to any network system and anycomputing device or system in which it is desirable to implement animage fusion system.

Also, there are multiple ways to implement the same or similarfunctionality, e.g., an appropriate API, tool kit, driver code,operating system, control, standalone or downloadable software objects,etc. which enables applications and services to take advantage of thetechniques provided herein. Thus, embodiments herein are contemplatedfrom the standpoint of an API (or other software object), as well asfrom a software or hardware object that implements one or moreembodiments as described herein. Thus, various embodiments describedherein can have aspects that are wholly in hardware, partly in hardwareand partly in software, as well as in software.

As utilized herein, the term “or” is intended to mean an inclusive “or”rather than an exclusive “or.” That is, unless specified otherwise, orclear from the context, the phrase “X employs A or B” is intended tomean any of the natural inclusive permutations. That is, the phrase “Xemploys A or B” is satisfied by any of the following instances: Xemploys A; X employs B; or X employs both A and B. In addition, thearticles “a” and “an” as used in this application and the appendedclaims should generally be construed to mean “one or more” unlessspecified otherwise or clear from the context to be directed to asingular form.

Further, as used herein, the term “exemplary” is intended to mean“serving as an illustration or example of something.”

Illustrative embodiments have been described, hereinabove. It will beapparent to those skilled in the art that the above devices and methodsmay incorporate changes and modifications without departing from thegeneral scope of the claimed subject matter. It is intended to includeall such modifications and alterations within the scope of the claimedsubject matter. Furthermore, to the extent that the term “includes” isused in either the detailed description or the claims, such term isintended to be inclusive in a manner similar to the term “comprising” as“comprising” is interpreted when employed as a transitional word in aclaim.

What is claimed is:
 1. A system, comprising: a processor coupled tomemory storing computer-executable instructions that, when executed bythe processor, configure the processor to: obtain feature informationindicative of a pose of a feature in a first image of a first modality;obtain tracking information from a spatial tracker configured toidentify a position and orientation of an imaging device producingimages of a second modality; and determine when a pose of the imagingdevice matches a predetermined pose suitable to produce a second imageof the feature having a pose that matches a reference pose of thefeature based on at least one of the feature information or the trackinginformation.
 2. The system of claim 1, wherein the processor is furtherconfigured to update, responsive to movement of the imaging device, anindication of the pose of the feature in the second image produced bythe imaging device based on at least one of the tracking information orthe feature information.
 3. The system of claim 2, wherein the processoris further configured to: obtain a relationship between the pose of theimaging device in space and a pose of an imaging plane of the imagingdevice; and update the indication of the pose of the feature in thesecond image based on the relationship.
 4. The system of claim 2,wherein, to update the indication of the pose of the feature, theprocessor is further configured to reslice the first image along avirtual imaging plane.
 5. The system of claim 4, wherein the processoris further configured to determine the virtual imaging plane based onthe tracking information.
 6. The system of claim 4, wherein theprocessor is further configured to interpolate image data of the firstimage corresponding to the virtual imaging plane.
 7. The system of claim1, wherein the processor is further configured to identify a transformbetween the first image and the second image.
 8. The system of claim 7,wherein, to update the identified position of the feature, the processoris further configured to apply the transform to the feature information.9. The system of claim 1, wherein the feature information includes acontour of the feature in the first image.
 10. The system of claim 1,wherein the feature information specifies an orientation of the featurein the first image relative to a given orientation of the imagingdevice.
 11. The system of claim 1, wherein the processor is furtherconfigured to: display a live image acquired from the imaging device;and display an overlay on the live image based on the featureinformation.
 12. The system of claim 2, wherein the processor is furtherconfigured to: display a live image acquired from the imaging device;display an overlay on the live image based on the feature information;and update the overlay based on the indication of the pose of thefeature in the second image as acquired by the imaging device.
 13. Amethod, comprising: acquiring a live image of a first modality from animaging device; acquiring tracking information from a spatial trackerthat indicates a position and orientation of the imaging device; anddetermining when the imaging device is in a desired pose relative to afeature in the live image thereby establishing a relationship between apose of the imaging device to a pose of the feature in a second image ofa second modality, wherein determining the imaging device is in thedesired pose is based on at least one the tracking information orfeature information indicative of the pose of the feature in the secondimage.
 14. The method of claim 13, further comprising updating anindication of the feature in the live image, responsive to movement ofthe imaging device, based on at least one of the tracking information orthe established relationship.
 15. The method of claim 14, wherein theindication of the feature is initially generated based on featureinformation indicating the pose of the feature in the second image withrespect to the desired pose of the imaging device relative to thefeature.
 16. The method of claim 13, wherein the feature information isa contour.
 17. The method of claim 14, wherein updating the indicationof the feature further comprises reslicing the second image along avirtual imaging plane, wherein the virtual imaging plane is determinedbased on the tracking information.
 18. The method of claim 17, whereinreslicing further comprises interpolating image data of the second imagecorresponding to the virtual imaging plane.
 19. The method of claim 14,wherein updating the indication of the feature further comprising:determining a transform between the desired pose of the imaging deviceand the position and orientation of the imaging device provided by thetracking information; and applying the transform to the indication ofthe feature.
 20. A non-transitory, computer-readable storage mediumhaving stored thereon computer-executable instructions that configure aprocessor to: obtain feature information indicative of a feature in afirst image of a first modality and a pose of the feature in the firstimage with respect to a desired pose of an imaging device configured toproduce images of a second modality; obtain tracking information from aspatial tracker configured to track a position and orientation of theimaging device in space; and determine when the imaging device isaligned with the desired pose based on at least one of a live imageproduced by the imaging device, the feature information, or the trackinginformation.
 21. The non-transitory, computer-readable storage medium ofclaim 20, further storing instructions that configure the processor toupdate an indication of the feature in the live image based on at leastone of the tracking information or the feature information.
 22. Thenon-transitory, computer-readable storage medium of claim 20, furtherstoring instructions that configure the processor to display an initialindication of the pose of the feature in the live image, wherein theinitial indication is generated based on the feature information andprovides a guide to align the imaging device with the desired pose. 23.The non-transitory computer-readable storage medium of claim 21, furtherstoring instructions that configure the processor to reslice the firstimage along a virtual imaging plane determined based on the trackinginformation and interpolate the feature information along the virtualimaging plane to generate an updated indication of the pose of thefeature in the live image.
 24. The non-transitory, computer-readablestorage medium of claim 21, further storing instructions that configurethe processor to: determine a transform between the desired pose and theposition and orientation of the imaging device provided by the trackinginformation; and apply the transform to the feature information toupdate the indication of the pose of the feature in the live image.